Awareness, incentives, and quality control techniques that identify questionable
data are the best ways to control data collection problems associated with the
"consistency of definitions."

In a later section of these comments, we address the training needs to improve
consistency of definitions in federal and State data systems,

Cross-Sectional and Longitudinal Studies.

An over-interpretation of the strong support for longitudinal studies contained
in the synthesized papers could lead some individuals to conclude that
cross-sectional studies are not appropriate for generating NCES time series
data. Very few papers addressed cross-sectional designs.

Cross-sectional designs can provide useful time series data. The data are often
less expensive to collect than longitudinal data, because individual students do
not need to be traced. The data can be used to establish trend information and
to corroborate information collected from smaller longitudinal samples.

One caution is highlighted for emphasis. National Assessment does some things
extremely well in measuring student performance, much better than the
longitudinal studies which have relied on paper and pencil multiple choice type
items to measure student performance. Historically, National Assessment has not
relied entirely on the multiple choice question for the collection of
performance data and much additional information has been collected, for
example, in writing and mathematical problem solving.

If National Assessment and the longitudinal studies are combined, the best of
both should be used. This would result in a massive and expensive study, but
one whose value would exceed the value of the combined results of two or more
individual studies. But, the expense may prove to be the fatal flaw in the
combination strategy. As ways to reduce the expense are considered, the
strengths of the individual systems may be lost.

If saving dollars is the motivating factor for combining cross-sectional (NAEP)
and longitudinal studies, care should be exercised.  However, if the more
effective use of dollars is the motivating factor, serious consideration should
be given to combining the National Assessment and the longitudinal studies.

Before NCES reaches a decision on this important and complex design issue, we
recommend that they commission a collection of papers where the pro's and con's
would be identified and considered on the appropriateness of combining the
National Assessment and the longitudinal studies, e.g., the advantages and
disadvantages of alternative sampling and analytical design options, appropriate
ways to measure student performance relative to the response burden imposed on
the student and the school, and cost effectiveness.  Such papers could serve as
a basis to stimulate public debate and subsequent input into NCES before they
reach a final decision as to the design that is of the most benefit and least
cost for NAEP or NELS or their combination.
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